'Is this right?' or 'Is that wrong?': Evidence from Dynamic Eye-Hand Movement in Decision Making

Eye tracking and hand motion (or mouse) tracking are complementary techniques to study the dynamics underlying human cognition. Eye tracking provides information about attention, reasoning, mental imagery, but figuring out the dynamics of cognition is hard. On the other hand, hand movement reveals the hidden states of high-level cognition as a continuous trajectory, but the detailed process is difficult to infer. Here, we use both eye and hand tracking while the subject watches a video drama and plays a multimodal memory game (MMG), a memory recall task designed to investigate the mechanism of recalling the contents of dramas. Our experimental results show that eye tracking and mouse tacking provide complementary information on cognitive processes. In particular, we found that, when humans make difficult decisions, they tend to ask ’Is the distractor wrong?’, rather than ’Is the decision right?’.